Skip to content

fix signed overflow in calculate() and MathLib constant folding - #9030

Open
Nussu06 wants to merge 3 commits into
cppcheck-opensource:mainfrom
Nussu06:constant-folding-signed-overflow
Open

Nussu06 wants to merge 3 commits into
cppcheck-opensource:mainfrom
Nussu06:constant-folding-signed-overflow

Conversation

@Nussu06

@Nussu06 Nussu06 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

calculate() in lib/calculate.h, MathLib::add/subtract/multiply/mod and valueFlowEnumValue fold integer constants from the checked code in MathLib::bigint with plain signed arithmetic, so code like x = 0x7fffffffffffffff + 1;, A<0x8000000000000000 % -1> or enum E { A = 0x7FFFFFFFFFFFFFFE, B, C }; makes cppcheck itself hit signed integer overflow (UBSan reports in calculate.h, mathlib.cpp and valueflow.cpp, and the % -1 case can trap on x86-64). Added a calculateArithmetic helper that does +, - and * for signed integer types on the unsigned type and used it at those sites, and x % -1 now returns 0, so folded values are the same as before and only the undefined behaviour is gone. With this, testrunner is clean under ASan+UBSan with signed-integer-overflow enabled (cmake/dynamic_analyzer_options.cmake still disables it with a TODO); I added asserts to TestMathLib::calculate and TestValueFlow::valueFlowCalculations plus a new TestSymbolDatabase::enum21, and they fail under UBSan without the fix. There is no trac ticket for this yet, could you create one? I will update the title to "Fix #NNNNN (...)" once it exists.

@dmcppcheck

Copy link
Copy Markdown
Contributor

There should probably be a trac ticket for this PR. Please create it, or ask us to create it. Then the PR title should be updated.

@dmcppcheck dmcppcheck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

I found no problems in the changed code. Under a -fsanitize=signed-integer-overflow build, all the cases the PR names no longer overflow, and the folded values stay the same as before (they wrap). One related site is still left: ValueFlowAnalyzer::writeValue in lib/vf_analyzers.cpp:374 does value->intvalue += (inc ? 1 : -1); for ++/--. So long long x = 0x7fffffffffffffff; x++; still makes cppcheck overflow a signed integer. The PR does not touch this path, and its tests don't cover it, so the claim that it gets rid of this kind of undefined behaviour is not complete. The same calculateArithmetic helper could be used there too.

Testing

  • I built cppcheck and testrunner from pr/ with the normal Makefile. ./testrunner TestMathLib TestValueFlow TestSymbolDatabase: 555 tests, 0 failed.
  • I built a second copy of pr/ with CXXFLAGS="-O1 -g -fsanitize=signed-integer-overflow,undefined -fno-sanitize-recover=signed-integer-overflow" and ran it on small inputs, one line per file:
    • 0x7fffffffffffffff + 1, 0x7fffffffffffffff * 2, LLONG_MIN % -1 in an expression and as a template argument, enum E { A = 0x7FFFFFFFFFFFFFFF, B, C };, x += 1 / y *= 3 after x = LLONG_MAX, unary minus and / -1 of LLONG_MIN, a condition x == LLONG_MAX followed by x + 1, and a loop near LLONG_MAX: no UBSan reports.
    • With --debug, the folded values are as expected. + gives -9223372036854775808, * gives -2, and the enum gives B = LLONG_MIN and C = LLONG_MIN+1. (LLONG_MIN <=> 1) still folds to -1.
  • A case that still overflows (not part of the PR's changes):
    // sp/s9.cpp
    void k() { long long x = 0x7fffffffffffffff; x++; ++x; }
    /work/pr_ubsan/cppcheck --std=c++20 --enable=all -q sp/s9.cpp
    lib/vf_analyzers.cpp:374:29: runtime error: signed integer overflow: 9223372036854775807 + 1 cannot be represented in type 'long long int'
        #0 ValueFlowAnalyzer::writeValue(...) lib/vf_analyzers.cpp:374
        #1 ValueFlowAnalyzer::update(...) lib/vf_analyzers.cpp:837
    
  • I did not run the full testrunner under ASan+UBSan, did not test the Boost int128_t build, and did not build base under UBSan.

Automated review of 8c80299 by Claude. The pull request was built and tested locally.

@Nussu06

Nussu06 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Good catch on ValueFlowAnalyzer::writeValue. The ++/-- path in lib/vf_analyzers.cpp now goes through calculateArithmetic as well, so long long x = 0x7fffffffffffffff; x++; wraps to LLONG_MIN like before without the signed overflow. I added a case for it in TestValueFlow::valueFlowAfterAssign.

About the trac ticket: could you create one for this? I'll change the title to "Fix #NNNNN (...)" as soon as it exists.

@dmcppcheck

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved.

@dmcppcheck

Copy link
Copy Markdown
Contributor

Test results for commit fd6eddb (tools/test-my-pr.py, main compared to this PR):

Test: http://ec2-16-170-140-253.eu-north-1.compute.amazonaws.com/pr-9030/
packages: 1000
Differing warnings: 0 (+0 -0)
Timing: -0.0% (http://ec2-16-170-140-253.eu-north-1.compute.amazonaws.com/pr-9030/2026-11-15_16-33-57-fd6eddb9553e_timing.html)

Posted automatically by the cppcheck PR test runner. +N: warnings only with this PR, -N: warnings only with main. The AI review is written by Claude and can be wrong.

@Nussu06
Nussu06 force-pushed the constant-folding-signed-overflow branch from fd6eddb to 14439b1 Compare November 17, 2026 19:23
@Nussu06

Nussu06 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on main. The conflict was with #9006, which added the compound assignment labels to calculate(). +=, -= and *= now go through calculateArithmetic as well, so x += 1 at LLONG_MAX doesn't overflow either. The full testrunner passes and is clean under -fsanitize=signed-integer-overflow.

I still don't see a trac ticket for this. Could one of you create it? I'll update the title to "Fix #NNNNN (...)" once it exists.

@dmcppcheck dmcppcheck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

I found no new problems in the code the PR changes. The ++/-- site in vf_analyzers.cpp from the earlier review now goes through calculateArithmetic, and long long x = LLONG_MAX; x++; no longer overflows under UBSan. The folded values are the same as before: they wrap.

Two signed overflows are still left in code the PR does not touch, so the claim that the overflow is gone is not complete:

  • Unary minus in lib/programmemory.cpp:2107 (v.intvalue = -v.intvalue;), hit by long long m = LLONG_MIN; long long n = -m;.
  • #if arithmetic in externals/simplecpp/simplecpp.cpp:1298, hit by #if 0x7fffffffffffffff + 1 < 0.

Neither line is in the diff, so I can't comment on them inline.

Comments

# Comment By Where Status
1 PR needs a trac ticket and an updated title @dmcppcheck general ⚠️ Open
2 ++/-- in ValueFlowAnalyzer::writeValue still overflows a signed integer AI review general ✅ Fixed

@Nussu06, please fix, resolve or reply to the open and new comments: #1.

Testing

  • Normal build: I built cppcheck and testrunner in pr/ with the normal Makefile.

    • ./testrunner TestMathLib TestValueFlow TestSymbolDatabase: 555 tests, 0 failed.
    • Full ./testrunner -q: 5420 tests, 0 failed.
  • UBSan build: I built a second copy of pr/ with CXXFLAGS="-O1 -g -fsanitize=signed-integer-overflow -fno-sanitize-recover=signed-integer-overflow -std=c++11". I ran it on files that cover:

    • x++ / --x at LLONG_MIN and LLONG_MAX
    • +=, *= and -= past the limits
    • an enum with B = A - 1 and A = LLONG_MIN
    • template arguments LLONG_MAX + 1, LLONG_MIN % -1 and LLONG_MAX * 3
    • <=>, a loop near LLONG_MAX, and conditions a == LLONG_MAX followed by a + 1 / a * 2
    • reverse analysis (x++; if (x == LLONG_MIN))
    • unsigned long long wrapping and int overflow

    None of the code the PR changes gave a UBSan report. The existing integerOverflow warning for A - 1 in the enum still comes.

  • Remaining overflows outside the diff (both are in code the PR does not change):

    // scratch/u2.cpp
    void h() {
      long long m = -0x7fffffffffffffff - 1;
      long long n = -m;
    }

    /work/pr_ub/cppcheck -q u2.cpp

    lib/programmemory.cpp:2107:38: runtime error: negation of -9223372036854775808 cannot be represented in type 'long long int'; cast to an unsigned type to negate this value to itself
    
    // scratch/t5.c
    #if 0x7fffffffffffffff + 1 < 0
    int a;
    #endif

    /work/pr_ub/cppcheck -q --enable=all t5.c

    externals/simplecpp/simplecpp.cpp:1298:20: runtime error: signed integer overflow: 9223372036854775807 + 1 cannot be represented in type 'long long int'
    
  • Not done: I did not build testrunner under UBSan, did not test the Boost int128 configuration, and did not build base/.


Automated review of 14439b1 by Claude. The pull request was built and tested locally.

@Nussu06

Nussu06 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

The unary minus in the programmemory Executor (lib/programmemory.cpp) now uses calculateArithmetic too, so -m with m == LLONG_MIN keeps the wrapped LLONG_MIN value without the overflow. I added a case to TestValueFlow::valueFlowAfterAssign, and it traps under -fsanitize=signed-integer-overflow without the fix.

The #if arithmetic one is in externals/simplecpp/simplecpp.cpp. Since that's vendored, I think it should be fixed upstream in simplecpp and not patched here, so I've left it out of this PR.

There's still no trac ticket for this. Could one of you create it? I'll change the title to "Fix #NNNNN (...)" once it exists.

@dmcppcheck dmcppcheck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

I found no problems in the lines the PR changes. They fold values the same way as before, and the new tests pass. But the PR still says it removes this kind of overflow, and two signed overflows are left that no earlier review mentioned. I reproduced both with a -fsanitize=signed-integer-overflow build:

  • Prefix ++/-- used inside an expression. For example long long y = ++x; with x == LLONG_MAX, or --x with x == LLONG_MIN. This goes through setTokenValue in lib/vf_settokenvalue.cpp:685/:694 and :715/:724 and still overflows. It is the sibling of the x++; statement path that the PR fixed in vf_analyzers.cpp.
  • Symbolic solving in solveExprValue. In lib/valueflow.cpp:6287 (value.intvalue -= intval;) and :6296 (value.intvalue += intval;), two chained +/- steps with large constants overflow.

Neither file is touched by the diff, so I have attached these remarks to the closest changed lines.

Comments

# Comment By Where Status
1 PR needs a trac ticket and an updated title @dmcppcheck general 💬 Answered: Author asks the maintainers to create the ticket; the title is not updated yet
2 ++/-- in ValueFlowAnalyzer::writeValue still overflows AI review general ✅ Fixed
3 Unary minus in programmemory.cpp still overflows AI review general ✅ Fixed
4 #if arithmetic in simplecpp still overflows AI review general 💬 Answered: Vendored code; should be fixed upstream in simplecpp
5 Prefix ++/-- in an expression still overflows in vf_settokenvalue.cpp AI review lib/vf_analyzers.cpp:374 🆕 New
6 solveExprValue (valueflow.cpp:6287/6296) still overflows AI review lib/calculate.h:54 🆕 New

@Nussu06, please fix, resolve or reply to the open and new comments: #5, #6.

Testing

  • Normal build: I built cppcheck and testrunner in pr/ with the Makefile. ./testrunner TestMathLib TestValueFlow TestSymbolDatabase TestProgramMemory ran 571 tests, 0 failed. I did not run the full testrunner.

  • UBSan build: I built a copy of pr/ with CXXFLAGS="-O1 -g -std=c++11 -fsanitize=signed-integer-overflow -fno-sanitize-recover=signed-integer-overflow" (into /work/pr_ub) and ran it on small files in scratch/. These gave no UBSan report:

    • x += 2 and a + 1 at LLONG_MAX
    • comparisons against LLONG_MAX and LLONG_MIN
    • -(m) with m == LLONG_MIN
    • a / -1 and a % -1 at runtime
    • template arguments LLONG_MIN % -1 and LLONG_MAX + 1
    • the enum X = LLONG_MIN, Y = X - 1

    The template argument LLONG_MIN / -1 still gives internalError: Division overflow. That is the existing behaviour from #4520.

  • Overflows that are still there:

    1. ++x used as an expression:
      void f() {
          long long x = 0x7fffffffffffffff;
          long long y = ++x;
      }
      /work/pr_ub/cppcheck -q a2.cpp
      lib/vf_settokenvalue.cpp:685:83: runtime error: signed integer overflow: 9223372036854775807 + 1 cannot be represented in type 'long long int'
      
      The same with long long x = -0x7fffffffffffffff - 1; long long y = --x;:
      lib/vf_settokenvalue.cpp:715:83: runtime error: signed integer overflow: -9223372036854775808 - 1 cannot be represented in type 'long long int'
      
    2. Symbolic solving, + case:
      void f(long long x) {
          long long y = x + 0x7fffffffffffffff;
          long long z = y + 0x7fffffffffffffff;
          if (z == 3) {}
      }
      /work/pr_ub/cppcheck -q a5.cpp
      lib/valueflow.cpp:6287:28: runtime error: signed integer overflow: -9223372036854775804 - 9223372036854775807 cannot be represented in type 'long long int'
      
    3. Symbolic solving, - case:
      void f(long long x) {
          long long y = x - 0x7fffffffffffffff;
          long long z = y - 0x7fffffffffffffff;
          long long w = z - 0x7fffffffffffffff;
          if (w == 0) {}
          if (x == 0) {}
      }
      /work/pr_ub/cppcheck -q a7.cpp
      lib/valueflow.cpp:6296:32: runtime error: signed integer overflow: 9223372036854775807 + 9223372036854775807 cannot be represented in type 'long long int'
      
  • Not done: I did not build base/ under UBSan. These lines are the same in base/, so the overflows are not regressions, only gaps in what the PR covers.


Automated review of 89183e7 by Claude. The pull request was built and tested locally.

Comment thread lib/vf_analyzers.cpp
if (d == Direction::Reverse)
inc = !inc;
value->intvalue += (inc ? 1 : -1);
value->intvalue = calculateArithmetic<MathLib::bigint>(inc ? '+' : '-', value->intvalue, 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

This fixes the x++; statement path, but prefix ++x/--x used inside an expression goes through setTokenValue in lib/vf_settokenvalue.cpp, and that still overflows. Lines 685/694 do v.intvalue + 1 and lines 715/724 do v.intvalue - 1.

void f() {
    long long x = 0x7fffffffffffffff;
    long long y = ++x;
}

Under -fsanitize=signed-integer-overflow this gives:

lib/vf_settokenvalue.cpp:685:83: runtime error: signed integer overflow: 9223372036854775807 + 1 cannot be represented in type 'long long int'

--x with x == LLONG_MIN gives the same error at line 715. Those sites could use calculateArithmetic too.

Comment thread lib/calculate.h
* x + y, x - y or x * y. Signed integer overflow is undefined behaviour, so for a signed integer
* type the operation is done on the unsigned type and the result wraps.
*/
template<class T>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

Another signed overflow is left that the PR does not cover: solveExprValue in lib/valueflow.cpp:6287 (value.intvalue -= intval;) and :6296 (value.intvalue += intval;).

void f(long long x) {
    long long y = x + 0x7fffffffffffffff;
    long long z = y + 0x7fffffffffffffff;
    if (z == 3) {}
}

Under UBSan this gives:

lib/valueflow.cpp:6287:28: runtime error: signed integer overflow: -9223372036854775804 - 9223372036854775807 cannot be represented in type 'long long int'

The same chain with three - 0x7fffffffffffffff steps hits line 6296. If the goal is a testrunner that is clean with the sanitizer turned on, these sites need the helper as well.

@dmcppcheck

Copy link
Copy Markdown
Contributor

Test results for commit 89183e7 (tools/test-my-pr.py, main compared to this PR):

Test: http://ec2-16-170-140-253.eu-north-1.compute.amazonaws.com/pr-9030/
packages: 1000
Differing warnings: 0 (+0 -0)
Timing: +0.0% (http://ec2-16-170-140-253.eu-north-1.compute.amazonaws.com/pr-9030/2026-11-18_18-26-49-89183e76f136_timing.html)

Posted automatically by the cppcheck PR test runner. +N: warnings only with this PR, -N: warnings only with main. The AI review is written by Claude and can be wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants